® 



J 



Europaisches Patentamt 
European Patent Office 
Office eu rope en des brevets 



® Publication number: 



0 493 893 A2 



® 



@ Application number: 91311339.5 
@ Date of filing: 05.12.91 



EUROPEAN PATENT APPLICATION 
© Int. CI 5: H04N 7/14 



® Priority: 31.1Z90 US 636324 

@ Date of publication of application: 
08.07.92 Bulletin 92/28 

® Designated Contracting States: 
DE FR GB IT 

® Applicant: AlVIERICAN TELEPHONE AND 
TELEGRAPH COMPANY 
550 IVIadison Avenue 
New Yoric, NY10022(US) 

@ Inventor: GItlin, Riciiard D. 
42 Windsor Drive 



Little Silver, New Jersey 07739(US) 

Inventor: Ribera, JoJin F, 

12 Glen Arden Drive 

Howell, New Jersey 07731 (US) 

Inventor: Woodworth, Clark 

145 Lynch Road 

Middletown, New Jersey 07748(US) 

© Representative: Johnston, Kenneth Graham et 
a) 

AT&T (UK) LTD. AT&T Intellectual Property 
Division 5 Mornlngton Road 
Woodford Green Essex, IG8 OTU(GB) 



< 

n 
cn 

00 
CO 

cn 



® Improved video display system. 

@ Direct eye-to-eye contact and compactness are 
achieved in a video display system including a cam- 
era positioned behind, or opposite the viewing side 
of, a display screen having first and second modes 
of operation. As a result, the camera and the person 
or persons viewing the display are on opposite sides 
of the display screen. The display screen is con- 
trolled to switch from the first or image display mode 
to the second or substantially transparent mode. 
When the display screen is in the substantially trans- 
parent mode, the camera is controlled to record 
images appearing on the viewing side of the screen. 
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Technical Field 

This invention relates to displays and, more 
particularly, to displays adapted for teleconferen- 
cing applications and the like wherein images of 
the calling and called persons viewing the displays 
are recorded and displayed on the called and call- 
ing displays, respectively. 

Background of the Invention 

Teleconferencing was introduced decades ago 
in a simplified form with picture telephones wherein 
bidirectional video and audio links were established 
between calling and called parties. With the advent 
of personal, desktop computers, teleconferencing 
has assumed a more complex form. Digital images 
such as text and graphics are displayed on each 
conferees display terminal while video images of 
the conferees are also displayed in a portion of the 
display. The latter display is made possible by 
positioning a camera to one side (top, bottom, left, 
right) of the display screen for recording images of 
the particular conferee viewing the display terminal. 
Since the conferee focusses attention on the 
screen and because the camera is positioned off to 
one side of the screen, eye contact is lacking 
between the conferee and the associated camera. 
This is known as a problem of parallax ™ a prob- 
lem which arises whenever a camera must view 
one or more persons watching a display screen. 
Parallax is also a problem in the broadcasting 
industry where text prompting devices are em- 
ployed. 

From the perspective of the camera suffering a 
parallax problem, the conferee appears to be doz- 
ing when the camera is above the screen, gazing 
to the left or right when the camera is right or left 
of the screen, or looking at the ceiling when the 
camera is below the screen. As the conferee's 
scrutiny of the screen display becomes more 
close, the problem of parallax becomes more no- 
ticeable. Eye contact with the camera establishes 
eye-to-eye contact with each of the conferees 
shown on the display screen thereby creating a 
feeling of interest among the conferees. Similarly, a 
lack of direct eye contact with the camera causes a 
lack of eye-to-eye contact with each of the con- 
ferees shown on the display screen which, in turn, 
creates a perception of disinterest or preoccupa- 
tion. 

While some teleconferencing display terminals 
continue to be produced with an inherent parallax 
problem, conventional solutions have been pro- 
posed. One such solution involves the combination 
of a CRT display with a side-mounted camera 
focussed on the conferee through a properly an- 
gled beam splitter. Each conferee can concentrate 



on the display screen while maintaining eye con- 
tact with the camera. Although such a display ter- 
minal conquers the parallax problem, it cannot be 
overlooked that the cost of success is quite high. 
5 This display terminal is extremely bulky covering 
an area several feet square. This is a significant 
percentage of a standard desk surface. Bulkiness is 
an inherent problem caused by the introduction of 
a beam splitter. 

Summary of the Invention 

Direct eye-to-eye contact and compactness are 
achieved in a video display system including a 

75 camera positioned behind, or opposite the viewing 
side of, a display screen having first and second 
modes of operation. As a result, the camera and 
the person or persons viewing the display are on 
opposite sides of the display screen. The display 

20 screen is controlled to switch from the first or 
image display mode to the second or substantially 
transparent mode. When the display screen is in 
the substan\ially transparent mode, the camera is 
controlled to record images appearing on the view- 

25 ing side of the screen. 

Compactness results from the use of a flat 
panel display such as liquid crystal or plasma. 
Further size reductions are possible by employing 
charge-coupled device (CCD) image sensors in the 

30 camera. 

For display screens requiring backlighting for 
viewing the displayed image, backlight illumination 
is provided together with the necessary control to 
extinguish or sufficiently lower the illumination level 

35 when the display screen is in the substantially 
transparent mode of operation. In another embodi- 
ment, a shutter provided between the camera lens 
and the display screen is controllably switched 
from a closed or reflective position to an open or 

40 transmissive position when the display screen en- 
ters the second mode of operation. 

Brief Description of the Drawing 

45 A more complete understanding of the inven- 

tion may be obtained by reading the following 
description of specific illustrative embodiments of 
the invention in conjunction with the appended 
drawing in which: 
50 FIG. 1 is a simplified block diagram of a bidirec- 
tional video display system; and 
FIGs. 2 and 3 show exemplary timing diagrams 
for the video display system in FIG. 1 . 

55 Detailed Description 

FIG. 1 shows an exemplary embodiment of the 
video display system realized in accordance with 
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the principles of the invention. The display system 
comprises display 1 , backlights 2. camera or image 
sensor 3, shutter 4, display processor 5, control 
circuit 6. and reflector 1 2. 

Display 1 comprises a substantially flat panel 
display element which has a transparent mode of 
operation and an "on" mode of operation. During 
the "on" mode of operation, a person can view 
images depicted on the display through front sur- 
face 13. During the transparent mode of operation, 
display 1 is sufficiently clear to allow light to pass 
through the display to camera 3 thereby allowing 
camera 3 to record an image of the person or 
persons viewing the display. 

Exemplary flat panel displays useful for realiz- 
ing display 1 include liquid crystal displays, plasma 
displays and ferro-electric displays. In general, the 
display is arranged as an active matrix of rows and 
columns including an active device for providing 
high contrast (ON/OFF) at each cross point. The 
cross point or intersection between a particular row 
lead and particular column lead corresponds to a 
pixel which is activated by energizing the particular 
row and particular column leads. Active devices 
commonly used at the cross points include thin 
film transistors (TFT), p-i-n diodes. MIM diodes and 
plasma devices. The latter device is understood by 
those skilled in the art to be different from the 
plasma display mentioned above. Since liquid cry- 
stal displays are not light emitting types of devices, 
the display would normally require some form of 
backlighting as shown, for example, by backlights 
2. It is contemplated that display 1 can provide 
monochrome, gray scale, or color display of the 
images. In an example from experimental practice, 
display 1 is realized as a TFT active matrix liquid 
crystal display such as the commercially available 
Sharp display model LQ14P01. 

Display 1 receives video images for display 
together with control signals via a set of leads 
shown simply as lead 7 from display processor 5. 
In some applications, it may be desirable to permit 
the display to be written in one part of the frame 
with a video image and subsequently to decay in 
the remaining portion of the frame to its transparent 
mode. In other applications, it may be desirable to 
write an image rapidly on display 1 and then, at a 
later time within the frame period, switch the dis- 
play to the transparent mode. Each pixel in display 
1 is written or activated once per frame period. For 
the latter application above, a video image can be 
accumulated in a frame buffer (not shown) of dis- 
play processor 5 so that the video image can be 
written to the pixels of display 1 within a small 
fraction of the frame period. 

As shown in FIG. 1, camera 3 is mounted 
behind display 1. It is shown adjacent to, and 
spaced apart from, back surface 14 of display 1. 



The distance from the camera to the back surface 
of the display is determined by the type of image 
sensor employed and any auxiliary focussing ap- 
paratus employed. When display 1 is switched into 
5 the transparent mode by a signal on lead 7 from 
display processor 5, camera 3 receives a start 
signal on lead 11 from control circuit 6 to begin 
recording images of the person or persons viewing 
the display. The video output signal of the viewer 
10 images recorded by camera 3 is shown on the lead 
labeled VIDEO OUT. 

In general, camera 3 Is mounted in a suffi- 
ciently centralized position behind the display to 
view persons watching the display thereby elimi- 
15 nating parallax by maintaining direct eye contact 
between the camera and the viewers. Camera 3 
must have sufficient lux sensitivity to maintain ac- 
ceptable video output of viewers situated at a nor- 
mal viewing distance from front surface 13 for 
20 display 1 when recording images in low levels of 
the available light. The transparency of display 1 
and shutter 4 can be optimized to provide very 
little loss of available light. Exemplary embodi- 
ments of the video display system have been re- 
25 alized and operated in the transparent mode with 
commercially available video cameras such as 
Sony models XC-71 1 and XC-007 (low lux rating). 
Additionally, standard CCD sensor arrays may he 
employed together with appropriate lens or focus- 
30 sing apparatus. For a 2/3 inch CCD image sensing 
array, a 10-1 5mm focal length lens apparatus pro- 
vides adequate viewing quality of the person read- 
ing text from the display screen. 

Synchronization between camera 3 and. display 
35 1 is maintained by a synchronizing circuit compris- 
ing display processor 5 and control circuit 6. The 
synchronizing circuit provides rapid time multiplex- 
ing between the transparent and "on" modes of 
display 1 to allow viewers to see the video images 
40 on display directly while permitting the camera to 
record images of the viewers. The synchronizing 
circuit receives video images destined for display 1 
on a lead labeled VIDEO IN. Control circuit 6 
develops fundamental synchronization from an ex- 
45 ternally supplied signal on the SYNC lead. Control 
circuit 6 utilizes standard circuits to manage opera- 
tion of the camera and shutter. Display processor 5 
converts the input video images to a form compati- 
ble with display 1 . Since display processor 5 con- 
so trols operation of display 1 , It Is convenient to have 
it also control operation of backlights 2. 

A synchronization signal is externally supplied 
on the SYNC lead from either camera 3 or display 
processor 5 or another device external to the dis- 
ss play system. This synchronization signal provides a 
reference for establishing timing Intervals based 
upon either the beginning of a recording period by 
camera 3 or the beginning of a video Image display 



5 



EP 0 493 893 A2 



6 



period by display 1, for example. Coordination be- 
tween control circuit 6 and display processor 5 Is 
established by signals on lead 9. This coordination 
is needed to insure that (1) the shutter and camera 
operations are synchronized with the transparent 
mode for the display and (2) the illumination occurs 
during the image display mode or "on" mode of 
operation for display 1. Other aspects of the syn- 
chronizing circuit are described below in relation to 
additional elements in the video display system. 

Duty cycles are defined, for camera 3, as the 
ratio between the time period for the transparent 
mode and the total frame period and, for display 1 , 
as the time period for the "on" mode of display 1 
and the total frame period. The duty cycle deter- 
mines the relative light levels received by camera 3 
and by a person viewing the display. In order to 
achieve acceptable contrast and brightness of dis- 
played video images for a viewer, display 1 is 
maintained in the "on" mode for a sufficiently long 
period of time. Similarly, the display is be main- 
tained in the transparent mode for a sufficient pe- 
riod of time to achieve proper light levels for the 
camera. Although a duty cycle of 50% is used in 
the examples described herein, it is contemplated 
that duty cycles other than 50% can be utilized. 
The frequency of mode switching for the display is 
determined by the display technology and the 
frame rate of the display. It is important to note that 
mode switching is generally synchronized with the 
display frame, camera operation depending on the 
persistence of the image sensor, operation of bac- 
klights, and operation of the shutter. Generally, the 
frame rate for the display is chosen to be 1/30 
second and, in a 50% duty cycle environment, the 
display is in the "on" mode for 1/15 second and in 
the transparent mode for 1/15 second. The frame 
rate is defined above for non-interlaced frames on 
display 1 . \Aftiere interlaced frames are desired, the 
rate of interest is the field rate wherein an odd and 
an even field comprise the frame. 

Shutter 4 is an optional element shown inter- 
posed between the image sensor (camera 3) and 
back surface 14 of display 1. Shutter 4. when 
employed, is useful for obscuring display images 
which could reach the image sensor in the "on" 
mode and for obscuring the camera lens aperture 
in reflector 12. In the latter case, this provides a 
uniform backlighting surface because, in the ab- 
sence of a shutter, an image of the open aperture 
could reach the viewer when display 1 is in the 
"on" mode. Shutter 4 is controlled to open and 
close by a signal on lead 1 0 from control circuit 6 
of the synchronizing circuit. Shutter 4 remains open 
at least for the time period that camera 3 is record- 
ing images of the viewer through transparent dis- 
play 1 . Shutter 4 closes in response to a signal on 
lead 10 coincident with display 1 entering the "on" 



mode, thereby providing a uniform backlighting 
surface with reflector 12. 

Typical shutters for use as shutter 4 include 
mechanical, optical and electrooptic effect devices. 
5 r^/lechanical shutters which permit electronic control 
provide sufficient blocking of the video images 
from the display to the image sensor but may lack 
the necessary speed of response in various ap- 
plications. Optical and electrooptic effect devices 

70 suitable for use as shutter 4 include liquid crystal 
devices acting as polarizers and Kerr effect de- 
vices. These devices respond to an applied signal 
to switch to an optically non-transmissive state 
whereas the devices may be normally transmissive 

/5 or transparent. The speed of response for each of 
these types of devices is well suited to the video 
display system described herein. Both latter de- 
vices are well known to those skilled in the art. 
While the shutter is simple to implement, it 

20 may be advantageous to avoid the use of a shutter 
and run the image sensor at a speed twice the 
frame rate, that is, 1/15 second, where an exem- 
plary standard frame rate is 1/30 second. In this 
type of operation, the image sensor generates a 

25 video output signal having the viewer images are 
. interleaved with display images. By selectively writ- 
ing only the viewer images Into an output frame 
buffer (not shown), it is possible to read the images 
out from the output frame buffer at the standard 

30 frame rate. CCD image sensing arrays are com- 
mercially available and can be combined with ap- 
propriate well known circuits to perform the higher 
speed image sensing. Output frame buffering with 
different writing and reading speeds is also well 

35 known in the art. 

Backlights 2 and reflective enclosure 12 pro- 
vide illumination of display 1 so that video images 
on the display can be viewed with appropriate 
brightness levels and contrast. Backlights 2 are 

40 optional elements. Moreover, it is contemplated 
that the backlights may be kept on during operation 
of the video display system without interfering with 
the operation of the image sensor for camera 3. 
Operation of backlights 2 is controlled to be either 

45 on or switched on and off by a signal on lead 8 
from display processor 5 of the synchronizing cir- 
cuit. For switched operation, cold cathode lamps 
are suitable for use as backlights 2 located gen- 
erally about the periphery of the display. Reflective 

50 enclosure 12 reflects the majority of light back 
toward the back surface of the display. The reflec- 
tive enclosure includes a small aperture through 
which the camera and optional shutter view the 
display and viewers. 

55 When backlight illumination of the display is 
employed, it may be desirable to reduce reflections 
from the back surface of display 1 toward camera 
3. To this end, a standard anti reflection coating or 
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antireflection film can be applied to back surface 
14. 

FIG. 2 shows a simplified timing diagram for 
exemplary operation of the video display system 
shown in FIG. 1. Operation of only the display 1. 5 
Image sensor or camera 3 and shutter 4 are con- 
sidered in FIG. 2. Backlights 2 are assumed to be 
on. It should be noted that the displays are not 
interlaced in this example. The video Image for 
frame i Is sent to display 1 beginning at time to- io 
The final pixel of the video image Is written to the 
display at time ti. Each pixel Is allowed to decay 
during the time interval designated frame i + 1 from 
time ti to time ta. Since the camera is generally 
positioned along a central axis substantially normal 75 
to and centered on the display, it Is necessary for 
the center portion of the display to become trans- 
parent before the camera begins operation. This 
occurs at time t2 when the camera receives a start 
signal. Recording of the viewer image continues 20 
until time t*, at which time the center pixels are 
again activated with another video display Image 
for frame i+2- From time tz until time t*. the 
shutter remains open. At time ta, the next video 
display image is written to display 1. A similar 25 
sequence of events occurs in each successive 
frame period. 

In this example, the video frame rate from the 
camera is twice the frame rate to the display be- 
cause the images sensed by the camera are output 30 
while the shutter is open and while the shutter Is 
closed. It is expected that this technique may pro- 
duce unacceptable levels of flickering on the dis- 
play. 

For displays where the decay time to a trans- 35 
parent mode for individual pixels is one-half of a 
frame period or less, the display can be operated 
at a normal frame rate. That is. display 1 can 
display every video frame i, frame i + 1, frame i + 2, 
etc. In order to accomplish this, it is necessary to 40 
write the video image to the display from a device 
such as frame buffer. This permits the display to 
begin displaying the image almost Instantaneously 
at the beginning of the frame period. By the middle 
of the frame period, the pixels begin to revert to a 45 
transparent mode. Recording of the viewer image 
commences three-quarters of the way through a 
frame period and continues through the first quarter 
of the next frame period. 

FIG. 3 shows a timing diagram for exemplary 50 
operation of the system in FIG. 1 wherein display 1 
has the capability for being cleared to a transparent 
mode or wherein display 1 has the capability to for 
individual pixels to decay rapidly into the transpar- 
ent mode. The displays described above can have 55 
Integral memory or separate frame buffers. In this 
example, the video display image is written to 
display 1 In the time period to to ti . From time ti to 



t2. the display Is actively displaying the image and 
the back illumination is also on. Also at ts, the 
Illumination is turned off. From time t2 to ta, the 
display Is allowed to decay to a transparent mode. 
Alternatively, the display receives a signal from the 
synchronization circuit causing all pixels to switch 
to a transparent mode until frame I + 1 Is written to 
the display at time t*. Once the display is In the 
transparent mode at time ta. the camera is ac- 
tivated by the start signal and the shutter is opened 
to allow recording of the viewer Images. A similar 
sequence of events occurs in each successive 
frame period. By maintaining the writing period (to 
to ti) and the clearing period (t2 to ta) as small as 
possible, light loss is minimized. 

For the example shown in FIG. 3, ferroelectric 
displays are attractive because they can be written 
rapidly and they retain the video image after writ- 
ing. Switching this type of display to the transpar- 
ent state is accomplished by rewriting the entire 
display or by clearing the display with the proper 
control signal. 

Claims 

1. A display system for providing video images to 
a viewer and for collecting Images of the view- 
er, the system characterized by 

display means (1) having first and second 
controllable modes of operation, the display 
means having a front (13) and back (14) sur- 
faces, the front surface for viewing the video 
images, 

image sensing means (3) for generating a 
video representation of the viewer image, the 
image sensing means being located adjacent 
to the back surface of the display means, and 

synchronizing means (5 to 11) connected 
to said display means for controlling the dis- 
play means to switch to the first mode of 
operation during a first time Interval so that the 
video image is displayed by the display 
means, 

the display means being controllable to 
switch to the second mode of operation during 
a second time interval so that viewer images 
are recorded by the image sensing means 
through the display means. 

2. The display system as defined in claim 1, 
characterized in that the synchronizing means 
serves to control the display means, and In 
that the first and second modes of operation 
are controllable. 

3. The display system as defined in claim 1 or 2, 
characterized in that the display means com- 
prises a substantially flat panel display, option- 
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ally selected from liquid crystal, plasma, and 
ferro-electric displays. 

The display system as defined in claim 1 , 2 or 
3. characterized in that the image sensor 
means includes a video camera (3), optionally 
including a charge-coupled device array. 

The display system as defined in claim 1,2,3, 
or 4, characterized in that the display system 
further includes a frame buffer for storing a 
next video Image, the frame buffer being re- 
sponsive to the synchronizing means for trans- 
ferring the stored next video image to the 
display means substantially at the beginning of 
the first time interval. 

The display system as defined in claim 1, 
further characterized by shutter means (4) in- 
terposed between the image sensing means 
and the back surface of the display means and 
having first and second modes of operation, 
the first mode of operation for scattering light 
impinging thereon and the second mode of 
operation for transmitting light therethrough so 
that images are transmitted to the image sens- 
ing means, and 

the synchronizing means (6) connected to 
the shutter means for controlling the shutter 
means to switch to the first mode of operation 
during the first time interval, and the synchro- 
nizing means for controlling the shutter means 
to switch to the second mode of operation 
during the second time interval. 

The display system as defined in any one 
preceding claim, further characterized by 

illumination means (2) positioned adjacent 
to the back surface of the display means for 
backlighting the display means. 
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10. A display system for providing video images to 
a viewer and for collecting images of the view- 
er, the system comprising, 

display means having first and second 
modes of operation, the display means having 
a front and back surfaces, the front surface for 
viewing the video images, 

image sensing means for generating a vid- 
eo representation of the viewer image, the 
image sensing means being located adjacent 
to the back surface of the display means, and 

synchronizing meaiS connected to said 
display means for controlling the display 
means to switch to the first mode of operation 
during a first time interval so that the video 
image is displayed by the display means, 

said display means automatically reverting 
to the second mode of operation during a 
second time interval so that viewer images are 
recorded by the image sensing means through 
the display means. 



8. The display system as defined in claim 7 
wherein the illumination means is switchably 
controllable to an "ON" state and to an "OFF" 
state, and 

the synchronizing means is connected to 
the illumination means for controlling the illu- 
mination means to switch to the "ON" state 
during substantially the first time interval, the 
synchronizing means for controlling the illu- 
mination means to switch to the "OFF" state 
during substantially the second time interval. 



45 



9. The display system as defined in claim 7 
wherein the shutter means includes an elec- 
trooptic device, such as a Kerr Cell, or in- 
cludes a liquid crystal cell. 
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